nvmeof: add volume cloning capability by gadididi · Pull Request #6277 · ceph/ceph-csi

gadididi · 2026-05-14T13:54:18Z

enable volume cloning support for nvmeof CSI driver by declaring CLONE_VOLUME capability. The RBD backend automatically handles all clone operations (snapshot-to-volume restore and volume-to-volume cloning) when VolumeContentSource is present in CreateVolume request.

The NVMeoF driver adds debug logging to track clone operations and creates namespaces pointing to the cloned RBD images. No changes to the NVMeoF gateway are required since clones are transparent at the block device level.

Need to test it!

Checklist:

Commit Message Formatting: Commit titles and messages follow
guidelines in the developer
guide.
Reviewed the developer guide on Submitting a Pull
Request
Pending release
notes
updated with breaking and/or notable changes for the next major release.
Documentation has been updated, if necessary.
Unit tests have been added, if necessary.
Integration tests have been added, if necessary.

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)

Copilot

Pull request overview

This PR starts enabling NVMe-oF volume cloning by advertising the CSI CLONE_VOLUME controller capability and passing clone-related CreateVolume requests through to the RBD backend, while adding debug logging for clone sources.

Changes:

Adds RPC_CLONE_VOLUME to the NVMe-oF controller capabilities.
Logs whether CreateVolume is using a snapshot or volume content source.
Documents that clone handling is delegated to the RBD backend.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
`internal/nvmeof/driver/driver.go`	Advertises NVMe-oF support for CSI volume cloning.
`internal/nvmeof/controller/controllerserver.go`	Adds clone-source debug logging before delegating `CreateVolume` to the RBD backend.

gadididi · 2026-05-17T06:02:09Z

 			csi.ControllerServiceCapability_RPC_MODIFY_VOLUME,
 			csi.ControllerServiceCapability_RPC_EXPAND_VOLUME,
 			csi.ControllerServiceCapability_RPC_CREATE_DELETE_SNAPSHOT,
+			csi.ControllerServiceCapability_RPC_CLONE_VOLUME,


I will add later.

Please include an e2e test for cloning with this PR, thanks!

I missed that. I am on it!

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

nixpanic · 2026-06-08T12:50:53Z

+			err = createPVCAndvalidatePV(f.ClientSet, sourcePVC, deployTimeout)
+			Expect(err).ShouldNot(HaveOccurred())
+
+			ginkgo.By("Binding source PVC to an application")


you don't need to bind the source volume to an app, that is validated in other tests already too.

ohh ok I can remove it. I thought you want both to be bind to app .
I will remove it after the test will be finish

Having both is ok, but not required. The cloned volume needs complete testing.

The current PR is fine by me, no need to update it unless there is something else you want to improve too.

ok, I will keep it as is.
looks like the cloning test passed!

�[1mSTEP:�[0m Creating a clone of the source PVC �[38;5;243m@ 06/08/26 13:16:31.494�[0m I0608 13:16:31.498376 79259 pvc.go:61] Waiting up to &PersistentVolumeClaim{ObjectMeta:{nvmeof-pvc-clone nvmeof-3536 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:VolumeResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{67108864 0} {<nil>} BinarySI},},},VolumeName:,Selector:nil,StorageClassName:*e2e-nvmeof-7592-sc,VolumeMode:nil,DataSource:&TypedLocalObjectReference{APIGroup:nil,Kind:PersistentVolumeClaim,Name:nvmeof-pvc,},DataSourceRef:nil,VolumeAttributesClassName:nil,},Status:PersistentVolumeClaimStatus{Phase:,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},AllocatedResourceStatuses:map[ResourceName]ClaimResourceStatus{},CurrentVolumeAttributesClassName:nil,ModifyVolumeStatus:nil,},} to be in Bound state I0608 13:16:31.498423 79259 pvc.go:64] waiting for PVC nvmeof-pvc-clone (0 seconds elapsed) I0608 13:16:33.499329 79259 pvc.go:64] waiting for PVC nvmeof-pvc-clone (2 seconds elapsed) I0608 13:16:33.503706 79259 pvc.go:86] PVC nvmeof-pvc-clone Event: Provisioning - External provisioner is provisioning volume for claim "nvmeof-3536/nvmeof-pvc-clone" I0608 13:16:33.503736 79259 pvc.go:86] PVC nvmeof-pvc-clone Event: ExternalProvisioning - Waiting for a volume to be created either by the external provisioner 'nvmeof.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. I0608 13:16:35.499332 79259 pvc.go:64] waiting for PVC nvmeof-pvc-clone (4 seconds elapsed) I0608 13:16:35.503453 79259 pv.go:468] Waiting for PV pvc-ff99cd28-102e-47fc-8d14-8aaec7c9a280 to bind to PVC nvmeof-pvc-clone I0608 13:16:35.503551 79259 pv.go:790] Waiting up to timeout=10m0s for PersistentVolumeClaims [nvmeof-pvc-clone] to have phase Bound I0608 13:16:35.505343 79259 pv.go:801] PersistentVolumeClaim nvmeof-pvc-clone found and phase=Bound (1.770669ms) I0608 13:16:35.505377 79259 pv.go:763] Waiting up to 10m0s for PersistentVolume pvc-ff99cd28-102e-47fc-8d14-8aaec7c9a280 to have phase Bound I0608 13:16:35.507083 79259 pv.go:771] PersistentVolume pvc-ff99cd28-102e-47fc-8d14-8aaec7c9a280 found and phase=Bound (1.688721ms) �[1mSTEP:�[0m Binding clone PVC to an application �[38;5;243m@ 06/08/26 13:16:35.51�[0m I0608 13:16:35.514333 79259 warnings.go:107] "Warning: would violate PodSecurity \"restricted:latest\": allowPrivilegeEscalation != false (container \"web-server\" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container \"web-server\" must set securityContext.capabilities.drop=[\"ALL\"]), runAsNonRoot != true (pod or container \"web-server\" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container \"web-server\" must set securityContext.seccompProfile.type to \"RuntimeDefault\" or \"Localhost\")" I0608 13:16:35.514428 79259 pod.go:377] Waiting up to 10m0s for csi-nvmeof-demo-pod-clone to be in Running state �[1mSTEP:�[0m Deleting the clone application and PVC �[38;5;243m@ 06/08/26 13:16:41.518�[0m

nixpanic · 2026-06-08T12:52:13Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

nixpanic · 2026-06-08T12:52:47Z

/test ci/centos/mini-e2e/k8s-1.35/nvmeof

Running at https://jenkins-ceph-csi.apps.ocp.cloud.ci.centos.org/job/mini-e2e_k8s-1.35-nvmeof/67/display/redirect

Madhu-1 · 2026-06-11T12:50:05Z

+	}
+
+	// Lock source volume to prevent concurrent deletion
+	if sourceVolumeID != "" {


lock on snapshot is not required?

Hi @Madhu-1 , I missed your comment here.
There is no need lock at nvmeof level for snapshot because the nvmeof snapshot call directly to rbd snapshot function..(and the rbd call has lock for that here- #6322). there are no more ops.
However, the cloning (which is running in CreateVolume()) call has nvmeof ops , so here need to use lock, prevent cloning + delete race

mergify · 2026-07-06T22:30:17Z

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

Pull request has been modified.

nixpanic · 2026-07-28T12:45:57Z

/queue

ceph-csi-bot · 2026-07-28T12:46:20Z

@Mergifyio rebase

mergify · 2026-07-28T12:47:43Z

rebase

🛑 The pull request rule doesn't match anymore

Details

This action has been cancelled.

enable volume cloning support for nvmeof CSI driver by declaring CLONE_VOLUME capability. The RBD backend automatically handles all clone operations (snapshot-to-volume restore and volume-to-volume cloning) when VolumeContentSource is present in CreateVolume request. The NVMeoF driver adds debug logging to track clone operations and creates namespaces pointing to the cloned RBD images. No changes to the NVMeoF gateway are required since clones are transparent at the block device level. Signed-off-by: gadi-didi <gadi.didi@ibm.com>

Add basic e2e test to validate PVC cloning functionality for nvmeof. Test creates a source PVC, clones it using DataSource, and verifies both can be deleted successfully. Signed-off-by: gadi-didi <gadi.didi@ibm.com>

ceph-csi-bot · 2026-07-28T12:48:18Z

/test ci/centos/k8s-e2e-external-storage/1.35

ceph-csi-bot · 2026-07-28T12:48:18Z

/test ci/centos/mini-e2e-helm/k8s-1.35

ceph-csi-bot · 2026-07-28T12:48:19Z

/test ci/centos/k8s-e2e-external-storage/1.34

ceph-csi-bot · 2026-07-28T12:48:19Z

/test ci/centos/mini-e2e/k8s-1.35

ceph-csi-bot · 2026-07-28T12:48:19Z

/test ci/centos/mini-e2e-helm/k8s-1.34

ceph-csi-bot · 2026-07-28T12:48:20Z

/test ci/centos/mini-e2e/k8s-1.34

ceph-csi-bot · 2026-07-28T12:48:24Z

/test ci/centos/k8s-e2e-external-storage/1.36

ceph-csi-bot · 2026-07-28T12:48:24Z

/test ci/centos/mini-e2e-helm/k8s-1.36

ceph-csi-bot · 2026-07-28T12:48:25Z

/test ci/centos/mini-e2e/k8s-1.36

ceph-csi-bot · 2026-07-28T12:48:25Z

/test ci/centos/upgrade-tests-cephfs

ceph-csi-bot · 2026-07-28T12:48:26Z

/test ci/centos/upgrade-tests-rbd

gadididi · 2026-07-29T06:56:53Z

/test ci/centos/mini-e2e/k8s-1.35

mergify · 2026-07-29T09:58:50Z

Deprecation notice: This pull request comes from a fork and was queued with update_method=rebase and update_bot_account impersonation. This capability will be removed on July 1, 2026. After this date, the merge queue will no longer be able to rebase fork pull requests with this configuration. To avoid disruption, switch to update_method=merge in your queue rule.

mergify · 2026-07-29T09:58:58Z

Merge Queue Status

✅ Entered queue — 2026-07-29 09:58 UTC · Rule: default · triggered by merge protections
✅ Checks skipped · PR is already up-to-date
✅ Merged — 2026-07-29 09:59 UTC · at 75e043f53bdb498c94ccf20b20bc335f732f05a3 · rebase

This pull request spent 26 seconds in the queue, including 5 seconds running CI.

Required conditions to merge

github-review-approved [🛡 GitHub branch protection]
- nvmeof: add volume cloning capability #6277

gadididi self-assigned this May 14, 2026

gadididi added the component/nvme-of Issues and PRs related to NVMe-oF. label May 14, 2026

gadididi requested review from Copilot and nixpanic and removed request for Copilot May 14, 2026 13:54

Copilot started reviewing on behalf of gadididi May 14, 2026 13:55 View session

Copilot AI reviewed May 14, 2026

View reviewed changes

gadididi marked this pull request as ready for review May 27, 2026 12:14

gadididi changed the title ~~[WIP] nvmeof: add volume cloning capability~~ nvmeof: add volume cloning capability Jun 3, 2026

This was referenced Jun 4, 2026

Race Condition During Concurrent Clone and Delete (CephFS & RBD) #6321

Open

Prevent concurrent DeleteVolume/DeleteSnapshot during CreateVolume #6322

Open

gadididi force-pushed the nvmeof/clone_feature branch from 7fa3701 to b17c26b Compare June 7, 2026 12:40

gadididi requested a review from Copilot June 7, 2026 13:57

Copilot started reviewing on behalf of gadididi June 7, 2026 13:57 View session

Copilot AI reviewed Jun 7, 2026

View reviewed changes

Comment thread e2e/nvmeof.go Outdated

Comment thread internal/nvmeof/controller/controllerserver.go

nixpanic reviewed Jun 8, 2026

View reviewed changes

Comment thread e2e/nvmeof.go Outdated

gadididi force-pushed the nvmeof/clone_feature branch from 30739de to 446ca9a Compare June 8, 2026 12:00

gadididi requested a review from Copilot June 8, 2026 12:02

Copilot started reviewing on behalf of gadididi June 8, 2026 12:02 View session

Copilot AI reviewed Jun 8, 2026

gadididi requested a review from nixpanic June 8, 2026 12:14

nixpanic previously approved these changes Jun 8, 2026

View reviewed changes

gadididi requested a review from a team June 8, 2026 14:49

Madhu-1 reviewed Jun 11, 2026

View reviewed changes

gadididi force-pushed the nvmeof/clone_feature branch from 446ca9a to 9ea69b7 Compare July 14, 2026 08:58

ceph-csi-bot added the queued/rebase label Jul 28, 2026

gadididi added 2 commits July 28, 2026 12:47

e2e: add clone test for NVMeoF driver

75e043f

Add basic e2e test to validate PVC cloning functionality for nvmeof. Test creates a source PVC, clones it using DataSource, and verifies both can be deleted successfully. Signed-off-by: gadi-didi <gadi.didi@ibm.com>

ceph-csi-bot force-pushed the nvmeof/clone_feature branch from 007bda5 to 75e043f Compare July 28, 2026 12:47

ceph-csi-bot added ok-to-test Label to trigger E2E tests and removed queued/rebase labels Jul 28, 2026

ceph-csi-bot added ci/in-progress/e2e This label acts like a guard and prevents Mergify from adding the `ok-to-test` label again. and removed ok-to-test Label to trigger E2E tests labels Jul 28, 2026

mergify Bot removed the ci/in-progress/e2e This label acts like a guard and prevents Mergify from adding the `ok-to-test` label again. label Jul 28, 2026

iPraveenParihar approved these changes Jul 29, 2026

View reviewed changes

gadididi mentioned this pull request Jul 29, 2026

nvmeof: add e2e test for snapshot #6435

Open

mergify Bot merged commit b17a575 into ceph:devel Jul 29, 2026
43 checks passed

gadididi deleted the nvmeof/clone_feature branch July 29, 2026 10:21

Uh oh!

Conversation

gadididi commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gadididi Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nixpanic commented Jun 8, 2026

Uh oh!

nixpanic commented Jun 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Jul 6, 2026

Uh oh!

nixpanic commented Jul 28, 2026

Uh oh!

ceph-csi-bot commented Jul 28, 2026

Uh oh!

mergify Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛑 The pull request rule doesn't match anymore

Uh oh!

ceph-csi-bot commented Jul 28, 2026

Uh oh!

ceph-csi-bot commented Jul 28, 2026

Uh oh!

ceph-csi-bot commented Jul 28, 2026

Uh oh!

ceph-csi-bot commented Jul 28, 2026

Uh oh!

ceph-csi-bot commented Jul 28, 2026

Uh oh!

ceph-csi-bot commented Jul 28, 2026

Uh oh!

ceph-csi-bot commented Jul 28, 2026

Uh oh!

ceph-csi-bot commented Jul 28, 2026

Uh oh!

ceph-csi-bot commented Jul 28, 2026

Uh oh!

ceph-csi-bot commented Jul 28, 2026

Uh oh!

ceph-csi-bot commented Jul 28, 2026

Uh oh!

gadididi commented Jul 29, 2026

gadididi commented May 14, 2026 •

edited

Loading

gadididi Jun 8, 2026 •

edited

Loading

mergify Bot commented Jul 28, 2026 •

edited

Loading

mergify Bot commented Jul 29, 2026 •

edited

Loading